Skip to content

Abbreviate verbose clickhouse logs.#10443

Open
jmcarp wants to merge 1 commit into
mainfrom
jmcarp/clickhouse-query-masking
Open

Abbreviate verbose clickhouse logs.#10443
jmcarp wants to merge 1 commit into
mainfrom
jmcarp/clickhouse-query-masking

Conversation

@jmcarp
Copy link
Copy Markdown
Contributor

@jmcarp jmcarp commented May 14, 2026

As of this writing, system.query_log is one of the largest clickhouse tables. From dogfood:

┌─database─┬─table──────────────────────────┬─compressed─┬─uncompressed─┬───────rows─┐
│ system   │ query_log                      │ 17.11 GiB  │ 39.49 GiB    │   24942352 │
│ oximeter │ measurements_cumulativeu64     │ 15.52 GiB  │ 91.91 GiB    │ 1576299396 │
│ oximeter │ measurements_f32               │ 14.52 GiB  │ 85.23 GiB    │ 1849126423 │
│ oximeter │ measurements_histogramu64      │ 2.76 GiB   │ 117.37 GiB   │   78425393 │
│ system   │ metric_log                     │ 742.29 MiB │ 12.84 GiB    │    2543307 │
└──────────┴────────────────────────────────┴────────────┴──────────────┴────────────┘

Most of this table's disk use is attributable to the query column, and most of that usage comes from the very long measurement queries run by oximeter, which take the form of:

SELECT * FROM oximeter.measurements_* WHERE timeseries_key IN (...)

The IN clause can include thousands of keys, and a single oximeter query can run multiple clickhouse queries of this form. This takes up a lot of space in the query log.

This patch truncates long IN (...) clauses. These aren't operationally useful, and shortening them shrinks system.query_log by about 80% in testing.

As of this writing, `system.query_log` is one of the largest clickhouse tables.
From dogfood:

```
┌─database─┬─table──────────────────────────┬─compressed─┬─uncompressed─┬───────rows─┐
│ system   │ query_log                      │ 17.11 GiB  │ 39.49 GiB    │   24942352 │
│ oximeter │ measurements_cumulativeu64     │ 15.52 GiB  │ 91.91 GiB    │ 1576299396 │
│ oximeter │ measurements_f32               │ 14.52 GiB  │ 85.23 GiB    │ 1849126423 │
│ oximeter │ measurements_histogramu64      │ 2.76 GiB   │ 117.37 GiB   │   78425393 │
│ system   │ metric_log                     │ 742.29 MiB │ 12.84 GiB    │    2543307 │
└──────────┴────────────────────────────────┴────────────┴──────────────┴────────────┘
```

Most of this table's disk use is attributable to the `query` column, and most
of that usage comes from the very long measurement queries run by oximeter,
which take the form of:

```
SELECT * FROM oximeter.measurements_* WHERE timeseries_key IN (...)
```

The `IN` clause can include thousands of keys, and a single oximeter query can
run multiple clickhouse queries of this form. This takes up a lot of space in
the query log.

This patch truncates long `IN (...)` clauses. These aren't operationally
useful, and shortening them shrinks `system.query_log` by about 80% in testing.
Copy link
Copy Markdown
Collaborator

@bnaecker bnaecker left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice fix, thanks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants